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-.Students'  responses  to  a  40-item  test  on  signed-number  subtraction 
are  viewed  as  consisting  of  two  different  components,  the  sign  and 
absolute  value  parts.  Each  component  is  scored  zero-one  for  wrong  or 
correct  of  the  corresponding  part  of  the  answer.  The  traditional 
scoring  yields  a  score  of  one  only  when  both  components  have  scores  of 
one.  By  taking  the  values  of  the  extended  caution  index  for  the 
absolute  value  component  as  the  x-axis  and  those  for  the  sign  component 
as  the  y-axis,  all  pairs  of  component  response  patterns  produced  by 
consistent  or  inconsistent  application  of  various  rules  or  random  errors 
are  mapped  into  the  ECI  product  space.  A  simulation  study  showed  that 
the  response  patterns  generated  by  changing  the  binary  score  of  any  one 
item  in  the  response  patterns  of  an  erroneous  rule  cluster  together  in 
the  ECI  product  space.  Moreover,  the  response  patterns  resulting  from 
the  same  kind  of  misconceptions  fall  closer  together  than  those 
resulting  from  very  different  kinds.  This  property  of  the  ECI  product 
space  opens  up  a  promising  way  to  handle  large  numbers  of  "bugs"  or 
rules  quantitatively.  But  the  ECls  are  not  defined  in  the  cases  of 
perfect  or  zero  scores.  Typically,  there  are  many  incomplete  rules 
(e.g.,  all  signs  of  the  responses  are  right  but  absolute  value  parts  are 
taken  by  some  erroneous  rule). 


Introduction 


Tatsuoka  and  Linn  (1982)  nave  recently  introduced  a  group  of  new 
indices,  extended  caution  indices  for  individual  i  (ECl^),  by  extending 
Sato's  original  caution  index  (1975)  into  the  context  of  item  response 
theory  (IRT).  The  caution  index  (Cj>)  is  designed  to  identify  anomalous 
binary  response  patterns  to  test  items  and  to  extract  information  not 
contained  in  the  total  score.  Of  course,  several  authors  have  developed 
appropriateness  indices  in  conjunction  with  IRT  (Wright,,  1977;  Levine 
and  Rubin,  1979),  but  Tatsuoka  and  Linn's  ECI*  has  a  unique  feature 
different  from  appropriateness  Indices.  When  estimating  the  parameters 
of  IRT  models  is  not  possible,  can  be  used  instead  of  the  ECIs, 
without  loss  of  conceptual  continuity.  The  item  response  curve,  and 
test  and  group  response  curves  used  in  defining  EC1  can  be  replaced  by 
standard  summary  statistics  based  on  observed  item  responses  such  as  the 
number  or  proportion  of  people  in  a  norm  group  answering  an  item 
correctly.  Cj  is  designed  for  using  such  standard  statistics  based  on 
sampling  theory.  Indeed,  Harnisch  and  Linn  (1981)  used  Cj  for  analyzing 
a  NAEP  dataset  (National  Assessment  of  Educational  Progress)  and 
successfully  diagnosed  curriculum  differences  and  school  differences 
within  a  school  district.  However,  Rudner  (1982)  found  that  IRT-based 
indices  detected  aberrant  response  patterns  more  efficiently  than  those 
based  on  summary  statistics.  A  recent  paper  by  Tatsuoka  and  Tatsuoka 
(1982c)  warns  that  there  may  be  an  upper  limit  to  any  personal  Indices' 
capability  to  detect  aberrant  response  patterns.  The  detection  rates  by 
ECIs  and  one  of  the  appropriateness  measures  are  about  60%  correct 
identification  of  aberrance,  and  20%  "false  alarms,"  l.e.,  normal 
response  patterns  labelled  aberrant.  The  result  agrees  with  kudner's 
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findings.  It  implies  Chat  further  Investigation  of  the  behaviors  of 
ECI8  may  be  needed. 

This  paper  introduces  a  new  application  of  ECIs  for  studying  a 
variety  of  students'  misconceptions,  which  produce  aberrant  response 
patterns.  By  so  doing,  we  may  be  able  to  uncover  a  different  aspect  of 
characteristics  of  ECIs.  When  tests  are  designed  to  measure  the  outcome 
of  learning  processes,  looking  into  a  whole  response  pattern  to  the  test 
items  often  provides  useful  information  to  diagnose  the  student's  state 
of  knowledge  (Birenbaum  &  Tatsuoka,  1980;  Birenbaum  &  Tatsuoka,  1982a). 
The  ECI  values  are  determined  by  using  response  patterns  and  provide 
desirable  information  for  diagnostic  purposes. 

An  error  diagnostic  system  for  signed  number  arithmetic  (SIGNBUG) 
has  been  developed  by  Tatsuoka  and  Baillie  (1982)  and  it  has 
successfully  diagnosed  quite  a  number  of  erroneous  rules  of  operation. 
Similar  diagnostic  system  for  arithmetic  such  as  whole  number 
subtraction  problems  (Brown  &  Burton,  1978)  have  also  found  hundereds  of 
erroneous  rules  resulting  from  incomplete  knowledge  or  some  kind  of 
misconception  ("bug")  possessed  by  the  students.  But  these  systems  are 
expensive  and  time  consuming  to  construct.  Besides,  they  can  be  used  in 
only  very  specific  domains  of  arithmetic.  Tatsuoka  and  Linn  (198/) 
briefly  discussed  using  one  of  the  five  ECIs  to  detect  the  erroneous 
rules  of  operation  in  signed-number  arithmetic.  The  Eds  have  two 
possible  advantages  over  previously  considered  approaches.  First, 
unlike  the  individual  consistency  index  (Tatsuoka  &•  Tatsuoka,  1981, 
1982b),  ECIs  do  not  require  repeated  measrues.  Second,  application  of 
ECIs  is  not  restricted  to  specific  content  domains  such  as  signed 
numbers  computations  or  whole  number  arithmetic.  Moreover,  the  number 
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of  erroneous  rules  can  sometimes  become  so  large  as  to  require  some 
quantitative  methods  to  classify  them  and  to  examine  their  relation¬ 
ships.  Tatsuoka  (1981)  has  tried  to  quantify  the  seriousness  of 
misconceptions  by  ascertaining  which  level  of  the  procedural  steps  it 
was  that  a  student  missed.  Her  approach  is  content  specific  aud  may  be 
useful  only  for  very  simple  problem  domains. 

This  paper  discusses  a  more  general  quantitative  approach  by 
utilizing  the  advantages  of  IRT-based  ECIs.  All  erroneous  rules  of 
operation  will  be  expressed  as  points  in  a  geometric  space  (called  "rule 
space").  Rule  space  is  useful  in  handling  large  numbers  of  bugs  and 
for  examining  their  psychometric  properties  such  as  "stability"  or 
"transitivity”  of  bugs  (Tatsuoka,  1982).  This  new  approach  will  be 
illustrated  with  the  test  data  obtained  from  a  40-item  test  of  signed- 
number  subtraction  problems.  Moreover,  the  relationships  between  rules 
and  their  partially  consistent  application  to  the  test  items  will  be 
illustrated  with  simulated  data  in  rule  space. 

Method  and  Procedure 

A  Brief  Introduction  of  the  Extended  Caution  Indices 

The  caution  index  for  subject  i  is  defined  as  the  complement  of  the 
ratio  of  two  covariances.  The  numerator  of  the  ratio  is  the  covariance 
of  observed  row  vector,  y^  »  (yil* • • • *yin)  *n  the  score  matrix  (yj,), 

[i  -  1,...,N  ,  j*l,...,n  where  N  is  the  number  of  subjects  and  n  the 
items],  and  the  column-sum  vector,  y  _  <y<1>  ym2,...,  y.n).  The 
denominator  is  the  covariance  of  the  corresponding  scores  rearranged  as 

g 

a  reverse  Guttman  vector,^  -  (ilj^,  Mi2»***»  Min)  atu*  column-sum 
vector  y#  .  Thus  Ci  is  given  by  Equation  (1). 
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The  values  of  ECIs  are  calculated  by  first  constructing  a  probability 

A 

matrix  with  elements  Pjj.  In  practice,  the  Pjj  can  be  replaced  by  Pij» 
whose  values  are  obtained  by  substituting  estimated  item  and  person 
parameters  in  the  logistic  function. 

One  of  the  ECIs,  ECI4,  is  defined  by  taking  the  ratio  of  two 
covariances  of  which  the  numerator  is  the  covariance  of  the  ith  row 
vector  in  the  score  matrix  (yij)  and  that  in  the  probability  matrix  (Pjj), 
which  are  denoted  by  and^,  respectively.  The  denominator  is  the 
covariance  of  the  column-sum  vector  of  (Pij)  which  is  denoted  by 

(G.i,  G.2 ,  •  •  • ,  G.n),  and^j.  The  following  Equation  (2)  is  the  fourth 
index  ECI4. 

(2)  ECI4  -  1  -  10V&  *&> 

cov^,^) 

The  second  ECI  is  ECI2  of  which  the  denominator  is  the  same  as  that  of 
ECI4,  but  the  numerator  is  the  covariance  of  ^  and  and  given  in  the 
following  equation  (3). 


covfyi  >  fi) 
ECI  2  -  1  -  _ —L 

cov^  ,  £) 


Unlike  the  caution  index,  the  numerator  of  ECI4  is  the  covariance  of 
the  observed  vector^  and  the  probability  vector  at  the  fixed  level  i. 
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which  is  not  a  group  dependent  vector.  As  a  result,  ECI4  should  be 
sensitive  to  the  anomalous  response  patterns  relative  to  the  anomaly  of 
response  patterns  in  comparison  with  the  row  vector^P^  at  the  level  ©i« 

On  the  other  hand,  the  identical  denominator,  (G^  ,  ^Pj_)  of  EC12  and  EC14 
can  be  considered  as  a  standardized  scaling  factor  ana  the  difference 
between  the  two  indices  comes  from  the  numerators  cov(^  ,  Pi)  and 
cov(^  ,  G) .  In  other  words,  the  numerator  of  ECI2  is  proportional  to 
the  cosine  of  the  angle  between  the  two  vectors  yj  ana  G  while  tne 
numerator  of  EC14  is  proportional  to  cosine  of  that  between  y^  and  P^  in 

'■Va  rsj1 

n  items  space.  Therefore,  the  difference  between  EC12  and  EC14  can  be 

said  to  lie  in  whether  the  response  pattern  of  the  observed  vector  y^ 

conforms  better  to  the  pattern  of  vector  Pi  or  that  of  the  group  average 

vector  G.  Tatsuoka  and  Linn  (1982)  demonstrated  briefly  that  ECI4  is 

moderately  effective  in  spotting  erroneous  rules  of  operation.  However, 

Eds  are  ©-dependent  measures  and  have  a  strong  tendency  to  give 

inflated  values  at  both  the  extremely  high  and  low  total  scores.  In 

order  to  avoid  the  undesired  property  of  ECIs,  Tatsuoka  and  Tatsuoka 

(1982c)  derived  the  expectations  and  variances  of  ECI4  and  EC12  and 

standardized  them.  The  standardized  ECIs  are  given  by  Equations  (4)  and  (5). 

ncov(Pi  -  yi  ,  Pi) 

(4)  ECI4z  -  - £1 - — -  y2 


ECI2Z 


ncov^Pi  -  jy.  ,  J3) 
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nent  response  patterns:  A  new  scoring  procedure 

i  40-item  free  response  test  that  comprises  four  parallel  subtests 
ined-number  subtraction  problems  was  administered  to  17/  eighth 
s  at  a  local  junior  high  school.  Tne  traditional  scoring  of  right 
mg  answers  was  decomposed  into  a  two-component  scoring  procedure 
olute-value  and  sign  parts  of  the  responses.  The  signs  of  the 
ises  to  n  items  were  scored  right  or  wrong  and  so  were  the  absolute 
Therefore,  a  regular  set  of  responses  to  n  items  was  decomposed 
wo  binary  response  patterns  related  to  the  sign  component  and  the 
te-value  component.  The  regular  response  patterns  are  element- 
lultiplicat ions  of  the  two  component  response  patterns.  Suppose  we 
hree  responses  to  10  items  resulting  from  the  following  four 
ous  rules. 

.ule  1 :  The  student  uses  the  right  rule  for  addition  problems, 
subtraction  problem,  he/she  changes  the  signs  of  the  number  in  the 
heses  first,  then  converts  the  subtraction  into  an  addition  problem 
he  right  conversion. 

-6  -  (-10)  =  -6  -  (+10)  *  -6  +  (-10) 
ule  2;  The  student  uses  a  wrong  rule  for  addition.  He/she  always 
cts  the  smaller  absolute  value  from  the  larger  absolute  value  and 
the  signs  of  the  first  number  in  the  answer.  The  student  converts 
iction  to  addition  problems  correctly,  then  consistently  applies 
me  erroneous  rule  to  the  new  addition  problem. 
ule  3:  The  addition  problems  are  answered  by  the  right  rule, 
iction  problems  are  converted  by  a  wrong  rule  —  by  simply  changing 
ieration  sign  minus,  -  ,  to  plus,  +  ,  except  for  L-S  (e.g.,  8-b) 

L  (e.g.,  6-8)  item  types.  The  student  knows  how  to  get  answers 


for  these  two  item  types  without  converting  them  to  addition  problems. 
He/she  uses  the  right  addition  rule  for  the  new  addition  problems  of  the 
other  eight  item  types. 

Rule  A:  The  student  always  subtracts  the  smaller  absolute  value 
from  the  larger  one  and  takes  the  sign  of  the  number  with  the  larger 
absolute  value  in  the  answer.  The  conversion  of  subtraction  problems  to 
addition  is  omitted  and  the  difference  between  addltiou  and  subtraction 
of  two  signed  numbers  seems  to  be  ignored. 

Table  1  summarizes  the  four  pairs  of  binary  vectors  and  responses 
yielded  by  the  four  rules.  As  can  be  seen  in  Table  1,  the  elementwise 

Insert  Table  1  about  here 

multiplications  of  the  two  component  score  vectors  yield  the  binary 
score  vector  of  regular  scoring.  The  response  patterns  scored  by  the 
regular  scoring  procedure  of  Rules  2  and  4  are  identical  but  the  si6n 
component  score  vectors  are  different.  Tat6uoka  and  Tatsuoka  (1981; 
showed  that  all  erroneous  rules  discovered  so  far  are  uniquely 
represented  by  the  two  component  score  vectors  with  the  10  items  of 
subtraction  problems.  Therefore,  the  two  component-response  patterns 
are  subjected  to  the  estimation  of  item  and  person  parameters  separately 
by  GETAB  (Baillie,  1980). 

Appendices  I  and  II  are  summary  lists  of  the  estimated  item 
parameters  for  the  two  sets  of  binary  response  patterns  obtained  from 


the  40-item  subtraction  test 


The  Binary  Response  Patterns  of  Three  Different  Scorings  Generated  by  Three  Erroneous  Rules 


Rule  Space 


A  rule  space  Is  defined  as  a  geometric  representation  of  the 

g 

rules  used  by  the  students.  Let  ECI4^Z,  i»l,...,N  be  the  values  of 
standardized  ECIs  obtained  from  the  sign-component  patterns  and  bCl4^z> 
i»l,...,N  be  from  the  absolute-value  component  patterns.  As  a  result,  a 
pair  of  two  real  numbers  is  associated  with  each  student.  However, 
since  ECIs  are  essentially  a  ratio  of  two  covariances,  they  caunot  be 
defined  when  the  scores  are  either  all  ones  or  all  zeros.  It  is 
impossible  to  assign  a  finite  number  to  the  response  patterns  yielded  by 
using  the  right  rule.  So  we  omit  the  students  who  answered  all  the 
items  right  or  all  the  items  wrong  in  this  study. 

A  plot  of  the  values  of  ECI*Z  (hereafter  the  i  will  be  omitted) 

against  the  absolute-component  true  scores  for  100  students  (excluding 
all  zeros  and  all  ones)  and  for  the  21  most  popular  erroneous  rules 
which  are  produced  by  a  variety  of  misconceptions,  is  given  in  Figure  1. 

Insert  Figure  1  about  here 

The  erroneous  rules  are  marked  by  a  small  circle  ”o"  while  the  real 
students  are  marked  by  "+M.  Each  point  in  Figure  1  represents  a 
absolute-component  response  pattern  for  the  40  items.  If  a  student 
responds  to  the  items  by  applying  erroneous  Rule  1  explained  in  Table  1 
consistently  throughout  the  test,  then  his  component  response  patterns 
yield  the  same  value  of  ECI4®  and  true  score  for  the  absolute  value 
component  as  those  produced  by  Rule  1  and  his  point  in  Figure  1 

coincides  with  the  point  of  Rule  1.  If  the  student  does  not  apply  his 

or  her  rule  perfectly  consistently  but  answers  one  or  two  items 
randomly,  then  his  or  her  component  response  pattern  doesn't  match  that 
produced  by  applying  the  rule  consistently.  The  values  of  the  true 
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scored  and  ECl4z  associated  with  the  student's  inconsistent  response 
patterns  do  not  coincide  with  the  values  yielaed  by  the  rule.  Given 
below  in  Figure  2  is  the  sign  component,  obtained  by  the  same  manner  as 
for  the  absolute-value  component  patterns. 

Insert  Figure  2  about  here 

As  can  be  seen  in  both  the  figures,  some  erroneous  rules 
represented  by  "o'*  are  found  near  the  x-axis  and  a  considerable  number 
of  aberrant  response  patterus  produced  by  real  students  "+"  is  seen 
outside  of  erroneous  rules  "oH.  It  yields  the  same  result  —  the  low 
detection  rates  of  aberrant  response  patterns  by  personal  Indices  —  as 
Rudner  (1982)  and  Tatsuoka  and  Tatsuoka  (1982a)  found  in  their  Btudies. 

Figure  3  is  a  plot  of  the  sign-component  true  scores  against 
absolute-value  component  true  scores.  The  cluster  near  the  top  right 
corner  in  Figure  3  represents  the  students  who  executed  the  right  rule 
for  responding  to  the  items  with  different  extents  of  consistency  as 
discussed  in  Tatsuoka  and  Tatsuoka  (1981). 

Insert  Figure  3  about  here 

The  ten  points  on  the  broken  line  perpendicular  to  the  x-axis  at 
©  ■  .292  in  Figure  3  have  the  same  sign  component  response  patterns.  It 
means  that  their  source  of  errors  may  be  Identical  with  respect  to 
understanding  of  the  absolute-value  operation  but  not  with  respect  to 
the  sign  operation.  For  example,  since  the  distance  of  the  two  points 
(rules  16  and  32)  is  very  small,  their  sources  of  misconception  may  be 
closely  related  with  one  another.  In  order  to  investigate  this 
question,  a  simulation  study  was  carried  out. 
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Plot  of  True  Score  for  Absolute  Value  Component  Against 
Sign  Component,  Real  Data  ("+")  and  Erroneous  Rules  ("0") 
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Procedures  for  Generating  Simulated  Inconsistent 
Responses  Around  a  Rule 

The  sign-component  pattern  of  Rule  1  given  in  Table  1  is 

-  (0101111111 j 

If  the  student  uses  Rule  1  inconsistently,  then  his/her  responses  to  the 
10  items  in  each  parallel  subtest  will  no  longer  match  the  response 
pattern  si.  Depending  on  his/her  degree  of  consistency,  possibly  one  or 
tvo  items  out  of  the  10  will  be  off  from  si  in  at  least  one  subtest.  We 
generate  10  vectors  in  each  of  which  exactly  one  element  (tne  1th; 

1*1,2, • • • ,10)  is  different  from  the  corresponding  element  of  si  (i.e., 
is  its  complement)  and  call  them  s^^j,  1*1,2, ...  ,10.  If  Rule  1  is 
consistently  applied  throughout  the  test,  then  the  four  response 
patterns  from  the  four  parallel  subtests  must  be  identical.  Since  a  few 
items  in  the  first  subtest  were  deleted  because  of  large  standard  errors 
of  estimate  by  the  maximum  likelihood  procedure  used  in  the  computer 
program  GETAB  (see  Appendix  I),  the  last  three  subsets  were  used  for 
generating  simulated  data  for  the  31  students  as  follows: 

The  first  "student's"  responses  to  the  four  parallel  blocks  of  10 
items  each  consist  of  four  replications  of ^  itself.  The  responses  of 
the  remaining  30  "students"  to  the  first  block  of  10  items  are  likewise 
^  without  modification.  However,  the  responses  to  the  second  through 
fourth  blocks  of  10  items  are  modified  for  the  first,  second  and  third 
subgroups  of  10  "students"  each  in  the  following  manner.  For  the  first 
subgroup  (i.e.,  "students"  2  through  11),  the  responses  to  the  second 
block  of  10  items  are  represented  by^(i ) » ^S,1(2) » •  •  •  *  ^(10) *  the 


third  and  fourth  blocks  remain  "responded  to"  by  Rule  1  to  yield  two 
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replications  of  s*.  For  Che  second  subgroup  (“students'*  12  through  21), 
the  responses  to  the  third  block  of  10  Items  become 
-5^(2) » *  *•  v^(10) »  while  those  t0  the  second  and  fourth  blocks  are  ^ 
Itself.  For  the  third  subgroup  ("students"  22  through  31),  Che  fourth- 
blocks  responses  become  ^(1)1 1*1 ,2 , . . .  ,10]  while  the  responses  to 
blocks  2  and  3  remain  as  s*.  Table  2  shows  the  40-element  response 
patterns  generated  for  the  31  "students,"  for  both  the  sign  and 
absolute-value  components.  The  31  ECI4  values,  including  the  perfect 
pattern  by  Rule  1  as  the  first  vector,  were  calculated. 

Insert  Table  2  about  here 

The  same  procedure  was  repeated  for  the  absolute-value  component 
patterns.  Thus,  two  sets  of  31  ECI4,  one  for  the  sign  component,  the 
other  for  the  absolute-value  component  patterns,  are  obtained.  FigureB 
Insert  Figures  4,  5  and  6  about  here 
4,  5,  and  6  show  that  the  30  non-cons is tent  (or  partially  consistent) 
response  patterns  plus  the  perfect  pattern  by  a  rule  cluster  together  no 
matter  which  axes  are  chosen  in  plotting.  This  implies  that  each 
erroneous  rule  has  in  its  vicinity  its  "non-consistent"  response 
patterns  —  the  responses  yielded  by  partially  consistent  application  of 
the  rule  —  and  they  may  form  a  unidimensional  set  of  points  like  the 
cluster  near  the  right  rule  in  Figure  3.  It  confirms  the  results 
demonstrated  in  Tatsuoka  and  Tatsuoka  (1981)  and  Birenbaum  and  Tatsuoka 
(1982a,  b),  which  have  Investigated  the  effects  of  misconceptions  on  the 
dimensionality  of  a  dataset  and  concluded  that  a  unidimensional  dataset 
in  signed-number  subtraction  problems  suggests  a  state  of  knowledge 
enabling  a  student  to  fairly  consistently  apply  the  right  rule  in 
responding  to  the  test  items.  This  can  be  interpreted  to  mean  that  a 
student  at  a  certain  state  of  knowledge  produces  a  particular  erroneous 


Plot  of  the  Four  Clusters  Consisting  of  Inconsistent  Responses 
Around  Rules  1,  2,  3  and  4  in  Table  I  for  Absolute  Value  Component. 
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Plot  of  the  Four  Clusters  Consisting  of  Inconsistent  Responses 
Around  Rules  1,  2,  3  and  4  in  Table  1  for  Sign  Component. 
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his  knowledge  level  rises,  he  starts  using  the  right  rule 
1  consistency  (Tatsuoka,  1982).  Finally,  his  responses 
and  more  consistent  and  reach  the  right  rule,  represented  by 
ht  corner  in  the  plot  in  Figure  3.  The  phenomenon  was 
served  in  several  datasets  collected  at  various  different 
ime  of  a  three  year  follow-up  study  of  signed-number 
which  is  summarized  in  Tatsuoka  (1982). 

cing  Figure  1  on  top  of  Figure  4,  we  are  able  to  find  a  few 
ose  absolute-value  component  patterns  fall  in  one  of  the  four 
The  error  analyses  on  these  responses  confirmed  that  they  are 
uced  by  applying  each  rule  with  partial  consistency, 
ur  erroneous  rules  given  in  Table  1  and  the  non-consistent 
eighboring  each  of  them  form  four  distinctly  different 
can  be  seen  in  Figures  4,  5  and  6.  However,  Rules  12  and 
ind  32  (for  a  more  detailed  description,  see  Tatsuoka  4 
>81)  marked  in  Figure  3  produce  only  two  clusters  as  seen  in 
Tet,  when  plotted  in  terms  of  the  absolute-value  and  sign 
four  distinctly  different  clusters  are  formed  in  the  rule 
d  by  the  absolute  value  true  scores  and  £<jI4^z  as  shown  in 


Insert  Figures  7  &  8  about  here 
nt  that  the  values  of  ECls  are  capable  of  separating 
terns  that  have  very  close  true  scores. 


Clusters  Around 


Clusters  Around  Rules  12,  16,  32  and  46  in  Figure 
:ore  Against  ECIS  . 


Summary  and  Discussion 


A  probabilistic  model  for  dealing  with  a  variety  of  misconceptions 
is  developed  and  its  useful  properties  are  demonstrated  with  a  40-item 
signed-number  subtraction  test.  The  model  is  tentatively  named  "rule 
space"  into  which  all  response  patterns  are  mapped.  Rule  space  is 
defined  as  a  cartesian  product  of  estimated  true  scores  and  the  values 
of  standardized  extended  caution  index,  ECI4Z  (Tatsuoka  &  Tatsuoka, 
1982a).  The  advantage  of  using  the  standardized  ECI  is  apparent  from 
Figures  1,  2,  4  and  5  because  EC1Z  has  the  property  of  dispersing 
response  patterns  at  the  same  fixed  0^  level.  Therefore,  if  two 
response  patterns  from  the  same  6  level  are  different,  then  their  £Cl^zs 
have  the  two  different  values.  As  can  be  seen  in  Table  1,  if  we 
decompose  the  regular  scoring  into  several  components  such  as  sign  and 
absolute-value  component  scores  in  signed  number  arithmetic,  then  each 
rule  has  a  much  greater  chance  to  be  represented  by  a  unique  set  of 
component  response  patterns.  In  the  study  of  signed  numbers,  all 
erroneous  rules  discovered  by  SIGNBUG  for  over  one  thousand  students 
have  been  uniquely  represented  so  far  by  the  two  sets  of  response 
patterns.  However,  each  subject  matter  may  require  a  unique 
consideration  of  scoring  procedures  for  the  rule-space  technique  to  be 
adapted.  Then,  by  forming  the  rule  spaces  it  may  be  possible  to 
determine  an  individual  student's  state  of  knowledge  by  identifying  a 
specific  misconception,  even  when  the  responses  are  only  partially 
consistent  and  cannot  be  diagnosed  by  the  SIGNBUG  approach. 
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Appendix  I 


Estimated  as  and  f>s  of  the  Sign  Component  Scores 
(N  -  172) 


Items 

/s 

as 

fts 

Items 

* 

as 

f>8 

11 

1.1499 

-1.0980 

33 

1.1922 

-.9672 

2 

.7023 

-1.1965 

34 

1.2023 

-.8493 

4* 

36 

1.3678 

-.9408 

6 

.8025 

-2.1632 

38 

1.0887 

-1.6538 

7* 

39 

.3135 

-4.9597 

8 

.8057 

-2.0711 

40 

.6888 

-1.9455 

9 

.7383 

-2.0964 

41 

.6432 

-2.2016 

12 

.4391 

-2.3949 

44 

.8380 

-1.8726 

13* 

45 

1.2010 

-  6620 

16 

1.1973 

-.4075 

48 

.8798 

-.4871 

17 

1.2428 

-1.1168 

49 

1.3178 

-.9092 

18 

.9571 

-.8658 

50 

1.3102 

-.8352 

20 

1.5489 

-1.0717 

52 

1.5050 

-.7836 

22 

.8465 

-2.0972 

54 

.9070 

-1.7964 

23 

.2113 

-6.8217 

55 

.3192 

-3.8657 

24 

.6632 

-1.9173 

56 

.8360 

-1.4982 

25 

.5425 

-2.4858 

57 

1.3123 

-1.4005 

28 

1.0400 

-1.6857 

60 

.6391 

-2.5071 

29 

1.3690 

-.8293 

61 

1.6311 

-.8136 

32 

1.0638 

-.5307 

64 

.6505 

-.5665 

*The  maximum  likelihood  procedure  did  not  converge 
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Appendix  II 


Estimated  as  and  tTs  of  Absolute  Value  Component  Scores 

(N  -  172) 


Items 

6 

a 

$ 

a 

Items 

A 

a 

a 

1? 

a 

1 

.2703 

-4.8475 

33 

.5805 

-2.1912 

2 

1.1097 

-.8180 

34 

1.4556 

-.4256 

4 

1.4449 

-.5630 

36 

1.5291 

-.4390 

6 

.9975 

-.7687 

38 

2.6000 

-.5729 

7 

.5672 

-2.0084 

39 

.6082 

-1.5998 

8* 

40 

.4443 

-2.6204 

9 

1.2718 

-.4632 

41 

1.2948 

-.5126 

12 

1.3016 

-.5674 

44 

1.4394 

-.6516 

13 

2.0761 

-.6438 

45 

1.4138 

-.6880 

16 

.5965 

-1.7412 

48 

.4394 

-1.7471 

17 

.5658 

-1.9440 

49 

.4538 

-2.3231 

18 

.9642 

-.6542 

50 

1.7342 

-.5412 

20 

1.4535 

-.6188 

52 

2.0177 

-.5074 

22 

2.6207 

-.4121 

54 

1.6602 

-.6386 

23 

.4755 

-2.3936 

55 

.6428 

-1.8268 

24 

.3909 

-2.2248 

56 

.4285 

-2.3563 

25 

2.1031 

0.3970 

57 

1.4859 

-.3948 

28 

1.4988 

-.9005 

60 

1.5030 

-.6680 

29 

1.9786 

-.6244 

61 

1.5419 

-.7219 

32 

.6542 

-1.6818 

64 

.4339 

-1.9638 

*The  maximum  likelihood  procedure  did  not  converge 
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